{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jFZKt6cqrBQ_"
      },
      "source": [
        "# TensorRT Optimization for Mask R-CNN"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lYcFFUPsrWwr"
      },
      "source": [
        "## Overview"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wwxSwqwKrXvf"
      },
      "source": [
        "This notebook outlines the process and results of converting a TensorFlow saved model developed with Mask R-CNN architecture into an optimized TensorRT model. The primary objective of this conversion is to enhance the inference speed on both edge devices and cloud infrastructure, thereby facilitating real-time application requirements and scalable deployment scenarios."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n2EV0Abhrac7"
      },
      "source": [
        "## Background"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y9ZOWXbOrcZC"
      },
      "source": [
        "The Mask R-CNN model, renowned for its efficiency in instance segmentation tasks, was initially trained using a high-quality dataset to identify and segment objects within images. Although the model achieved a high accuracy, its inference time on standard hardware was a considerable bottleneck, taking approximately 35 seconds per image."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "txhQMEv5riDY"
      },
      "source": [
        "## Objective"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hK2IyrLwrlg8"
      },
      "source": [
        "To significantly reduce the inference time of the 2 Mask R-CNN model without compromising its accuracy, ensuring it meets the latency requirements of real-time applications. The model should be capable of delivering prompt predictions on edge devices with limited computational resources as well as on cloud platforms.\n",
        "\n",
        "The TensorFlow saved model was converted into a TensorRT model using Tensorflow library."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kyYWvlyHsF1h"
      },
      "source": [
        "**Note:\n",
        "To execute this Colab notebook effectively, please ensure that you switch the runtime to utilize a GPU. Additionally, for optimal performance, select the 'High-RAM' option which is available under the 'Runtime' tab at the top of the Colab notebook interface. This configuration is essential for handling compute-intensive operations and large datasets without running into memory constraints.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GQ_GovZfQe4p"
      },
      "source": [
        "## Download required files \u0026 scripts."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uK_Bk5pD6vwz"
      },
      "outputs": [],
      "source": [
        "!apt-get install tensorrt uff-converter-tf"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "v_nKwP0wPoOL"
      },
      "outputs": [],
      "source": [
        "# Download preprocessing script.\n",
        "url = (\n",
        "    \"https://raw.githubusercontent.com/\"\n",
        "    \"tensorflow/models/master/\"\n",
        "    \"official/projects/waste_identification_ml/\"\n",
        "    \"model_inference/preprocessing.py\"\n",
        ")\n",
        "\n",
        "!wget -q {url}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VFoknp3ZsgUL"
      },
      "outputs": [],
      "source": [
        "# Download the script to pull instance segmentation model weights from the\n",
        "# TF Model Garden repo.\n",
        "url = (\n",
        "    \"https://raw.githubusercontent.com/\"\n",
        "    \"tensorflow/models/master/\"\n",
        "    \"official/projects/waste_identification_ml/\"\n",
        "    \"model_inference/download_and_unzip_models.py\"\n",
        ")\n",
        "\n",
        "!wget -q {url}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ArBZijNKtL8v",
        "outputId": "6793decd-1644-4f75-d2f5-b4d206ddd6fb"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
            "                                 Dload  Upload   Total   Spent    Left  Speed\n",
            "100 3303k  100 3303k    0     0  2120k      0  0:00:01  0:00:01 --:--:-- 2120k\n",
            "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
            "                                 Dload  Upload   Total   Spent    Left  Speed\n",
            "100 1913k  100 1913k    0     0   942k      0  0:00:02  0:00:02 --:--:--  943k\n"
          ]
        }
      ],
      "source": [
        "# download the sample image from the circularnet project\n",
        "url1 = (\n",
        "    \"https://raw.githubusercontent.com/tensorflow/models/master/official/\"\n",
        "    \"projects/waste_identification_ml/pre_processing/config/sample_images/\"\n",
        "    \"image_2.png\"\n",
        ")\n",
        "\n",
        "url2 = (\n",
        "    \"https://raw.githubusercontent.com/tensorflow/models/master/official/\"\n",
        "    \"projects/waste_identification_ml/pre_processing/config/sample_images/\"\n",
        "    \"image_4.png\"\n",
        ")\n",
        "\n",
        "!curl -O {url1}\n",
        "!curl -O {url2}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "olZx5rVUQo5c"
      },
      "source": [
        "## Import required packages."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "McVjuDBCvdGb",
        "outputId": "1b097eea-5467-4f05-84d7-e2890e0f4ef6"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m7.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Building wheel for tensorrt (setup.py) ... \u001b[?25l\u001b[?25hdone\n"
          ]
        }
      ],
      "source": [
        "!python3 -m pip install -q -U  tensorrt tf_keras"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WSX-u14IvY-6",
        "outputId": "16377c07-6190-4b30-a83a-9775df0863a7"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "8.6.1\n"
          ]
        }
      ],
      "source": [
        "import tensorrt\n",
        "print(tensorrt.__version__)\n",
        "assert tensorrt.Builder(tensorrt.Logger())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4RuDEk9WtonR"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import sys\n",
        "\n",
        "from PIL import Image\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "from six import BytesIO\n",
        "from six.moves.urllib.request import urlopen\n",
        "from typing import Any, Callable\n",
        "import preprocessing\n",
        "\n",
        "import logging\n",
        "logging.disable(logging.WARNING)\n",
        "\n",
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LOpbotPFOaFv"
      },
      "source": [
        "## Utils"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YSbWKz1Atliw"
      },
      "outputs": [],
      "source": [
        "def load_image_into_numpy_array(path: str) -\u003e np.ndarray:\n",
        "  \"\"\"Load an image from file into a numpy array.\n",
        "\n",
        "  Puts image into numpy array to feed into tensorflow graph.\n",
        "  Note that by convention we put it into a numpy array with shape\n",
        "  (height, width, channels), where channels=3 for RGB.\n",
        "\n",
        "  Args:\n",
        "    path: the file path to the image\n",
        "\n",
        "  Returns:\n",
        "    uint8 numpy array with shape (1, h, w, 3)\n",
        "  \"\"\"\n",
        "  image = None\n",
        "  if(path.startswith('http')):\n",
        "    response = urlopen(path)\n",
        "    image_data = response.read()\n",
        "    image_data = BytesIO(image_data)\n",
        "    image = Image.open(image_data)\n",
        "  else:\n",
        "    image_data = tf.io.gfile.GFile(path, 'rb').read()\n",
        "    image = Image.open(BytesIO(image_data))\n",
        "\n",
        "  (im_width, im_height) = image.size\n",
        "  return np.array(image.getdata()).reshape(\n",
        "      (1, im_height, im_width, 3)).astype(np.uint8)\n",
        "\n",
        "\n",
        "def load_model(model_handle: str) -\u003e Callable:\n",
        "    \"\"\"Loads a TensorFlow SavedModel and returns a function that can be used to\n",
        "    make predictions.\n",
        "\n",
        "    Args:\n",
        "      model_handle: A path to a TensorFlow SavedModel.\n",
        "\n",
        "    Returns:\n",
        "      A function that can be used to make predictions.\n",
        "    \"\"\"\n",
        "    print('loading model...')\n",
        "    print(model_handle)\n",
        "    model = tf.saved_model.load(model_handle)\n",
        "    print('model loaded!')\n",
        "    detection_fn = model.signatures['serving_default']\n",
        "    return detection_fn\n",
        "\n",
        "\n",
        "def perform_detection(model: Callable, image: np.ndarray) -\u003e dict[str, Any]:\n",
        "  \"\"\"Performs Mask RCNN on an image using the specified model.\n",
        "\n",
        "  Args:\n",
        "    model: A function that can be used to make predictions.\n",
        "    image_np: A NumPy array representing the image to be detected.\n",
        "\n",
        "  Returns:\n",
        "    A list of detections.\n",
        "  \"\"\"\n",
        "  detection_fn = model(image)\n",
        "  detection_fn = {key: value.numpy() for key, value in detection_fn.items()}\n",
        "  return detection_fn\n",
        "\n",
        "\n",
        "def create_directory(path: str):\n",
        "    \"\"\"Create a directory at the specified path if it does not exist.\n",
        "\n",
        "    Args:\n",
        "        path (str): The path of the directory to create.\n",
        "    \"\"\"\n",
        "    try:\n",
        "        os.makedirs(path, exist_ok=True)\n",
        "        print(f'Directory {path} created successfully')\n",
        "    except Exception as e:\n",
        "        print(f'Failed to create directory {path}: {e}')\n",
        "\n",
        "\n",
        "def convert_to_tensorrt(\n",
        "    saved_model_dir: str,\n",
        "    output_saved_model_dir: str\n",
        "    ) -\u003e Callable:\n",
        "    \"\"\"\n",
        "    Converts a TensorFlow SavedModel to TensorRT format.\n",
        "\n",
        "    Args:\n",
        "      saved_model_dir: The directory where the original TensorFlow SavedModel is\n",
        "      stored.\n",
        "      output_saved_model_dir: The directory where the TensorRT-converted model\n",
        "      will be saved.\n",
        "\n",
        "    Returns:\n",
        "      Callable: A generator function that yields input data for building TRT\n",
        "      engines.\n",
        "    \"\"\"\n",
        "    params = tf.experimental.tensorrt.ConversionParams(\n",
        "    precision_mode='FP16',\n",
        "    # Set this to a large enough number so it can cache all the engines.\n",
        "    maximum_cached_engines=16\n",
        "    )\n",
        "\n",
        "    converter = tf.experimental.tensorrt.Converter(\n",
        "        input_saved_model_dir=saved_model_dir, conversion_params=params\n",
        "    )\n",
        "\n",
        "    converter.convert()\n",
        "\n",
        "    # Define a generator function that yields input data, and use it to execute\n",
        "    # the graph to build TRT engines.\n",
        "    def my_input_fn():\n",
        "      yield image1\n",
        "\n",
        "    converter.build(input_fn=my_input_fn)  # Generate corresponding TRT engines\n",
        "    converter.save(output_saved_model_dir)  # Generated engines will be saved.\n",
        "\n",
        "\n",
        "def process_image(image_path: str) -\u003e tf.Tensor:\n",
        "  \"\"\"\n",
        "  Processes an image from a given file path.\n",
        "\n",
        "  This function reads an image from the specified path, resizes it, and applies\n",
        "  normalization preprocessing.\n",
        "\n",
        "  Args:\n",
        "    image_path: The file path of the image to be processed.\n",
        "\n",
        "  Returns:\n",
        "    A TensorFlow Tensor representing the processed image.\n",
        "  \"\"\"\n",
        "  image_np = load_image_into_numpy_array(image_path)\n",
        "  image_np_cp = tf.image.resize(image_np[0], (512, 1024), method=tf.image.ResizeMethod.AREA)\n",
        "  image_np_cp = tf.cast(image_np_cp, tf.uint8)\n",
        "  image_np = preprocessing.normalize_image(image_np_cp)\n",
        "  image_np = tf.expand_dims(image_np, axis=0)\n",
        "  return image_np"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hy1CO7FysqE1"
      },
      "source": [
        "## Import both Mask RCNN saved model(material \u0026 material form) from the repo."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EH_s2raFsYfq"
      },
      "outputs": [],
      "source": [
        "# 'material_model' output is both material and its sub type e.g. Plastics_PET.\n",
        "# 'material_form_model' outputs the form of an object e.g. can, bottle, etc.\n",
        "MODEL_WEIGHTS = {\n",
        "    'material_url': (\n",
        "        'https://storage.googleapis.com/tf_model_garden/vision/'\n",
        "        'waste_identification_ml/two_model_strategy/material/'\n",
        "        'material_version_2.zip'\n",
        "    ),\n",
        "    'material_form_url': (\n",
        "        'https://storage.googleapis.com/tf_model_garden/vision/'\n",
        "        'waste_identification_ml/two_model_strategy/material_form/'\n",
        "        'material_form_version_2.zip'\n",
        "    ),\n",
        "}\n",
        "\n",
        "\n",
        "SAVED_MODEL_PATH = {\n",
        "'material_model' : 'material/saved_model/',\n",
        "'material_form_model' : 'material_form/saved_model/',\n",
        "}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hLTB2Q0Tsmcx"
      },
      "outputs": [],
      "source": [
        "# Download the model weights from the Google's repo.\n",
        "url1 = MODEL_WEIGHTS['material_url']\n",
        "url2 = MODEL_WEIGHTS['material_form_url']\n",
        "!python3 download_and_unzip_models.py $url1 $url2"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "euasOIi2tJHA"
      },
      "source": [
        "## Preprocess an image."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ozr7NLHbtTvM"
      },
      "outputs": [],
      "source": [
        "image1  = process_image('image_2.png')\n",
        "image2 = process_image('image_4.png')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oi6abxdeugCh"
      },
      "source": [
        "## Load original SavedModel."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bFZnUjV7uj_r",
        "outputId": "ac1818ab-bf34-42b3-c00e-50cd684779a9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "loading model...\n",
            "material/saved_model/\n",
            "model loaded!\n",
            "loading model...\n",
            "material_form/saved_model/\n",
            "model loaded!\n"
          ]
        }
      ],
      "source": [
        "# Loading both models.\n",
        "detection_fns = [\n",
        "    load_model(model_path)\n",
        "    for model_path in SAVED_MODEL_PATH.values()\n",
        "]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DRjqNjUUvZ7K"
      },
      "source": [
        "# Convert to TensorRT model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ONvJgKJjv0dw"
      },
      "outputs": [],
      "source": [
        "TENSORRT_MODEL_PATH = {\n",
        "'material_model' : 'tensorrt/material/saved_model/',\n",
        "'material_form_model' : 'tensorrt/material_form/saved_model/',\n",
        "}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zaeRIsVXv3vG",
        "outputId": "d5f0555b-29c2-4d87-b72c-848f67772994"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Directory tensorrt/material/saved_model/ created successfully\n",
            "Directory tensorrt/material_form/saved_model/ created successfully\n"
          ]
        }
      ],
      "source": [
        "# Create directories to store TensorRT models.\n",
        "for value in TENSORRT_MODEL_PATH.values():\n",
        "  create_directory(value)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HNLEo7BRv-db",
        "outputId": "189e9842-0bee-46ae-b0dc-8051d5a23774"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "material/saved_model/ tensorrt/material/saved_model/\n",
            "material_form/saved_model/ tensorrt/material_form/saved_model/\n"
          ]
        }
      ],
      "source": [
        "# Convert Tensorflow saved models into TensorRT models.\n",
        "for key in SAVED_MODEL_PATH.keys():\n",
        "    value1 = SAVED_MODEL_PATH.get(key)\n",
        "    value2 = TENSORRT_MODEL_PATH.get(key)\n",
        "    print(value1, value2)\n",
        "    convert_to_tensorrt(value1, value2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vsl1OIMnUpGm"
      },
      "source": [
        "## Load TensorRT models."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xq_8sCo5wX74",
        "outputId": "dd4a9bd1-4cd5-4798-f7d2-71b0d402861b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "loading model...\n",
            "tensorrt/material/saved_model/\n",
            "model loaded!\n",
            "loading model...\n",
            "tensorrt/material_form/saved_model/\n",
            "model loaded!\n"
          ]
        }
      ],
      "source": [
        "# Loading both models.\n",
        "detection_fns_tensorrt = [\n",
        "    load_model(model_path)\n",
        "    for model_path in TENSORRT_MODEL_PATH.values()\n",
        "]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aOuzd9BZD2uP"
      },
      "source": [
        "## Checking speed with SavedModel."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NyCPXNTHArMN",
        "outputId": "7e1f9354-7347-4d86-c831-92c35333d939"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "386 ms ± 1.43 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
          ]
        }
      ],
      "source": [
        "%%timeit\n",
        "# Inference speed with first image.\n",
        "results = list(\n",
        "    map(\n",
        "        lambda model: perform_detection(model, image1),\n",
        "        detection_fns\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UNU4CTIaA3oA",
        "outputId": "2e82a72a-4d68-45ca-a269-a4a22c043451"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "169 ms ± 1.13 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
          ]
        }
      ],
      "source": [
        "%%timeit\n",
        "detection_fns[0](image2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ecJW0MU4BGF_",
        "outputId": "009107b8-28c4-40d8-dd58-8a486dfcc7b0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "200 ms ± 1.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
          ]
        }
      ],
      "source": [
        "%%timeit\n",
        "detection_fns[1](image2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sPK4-NdBwT3R"
      },
      "source": [
        "## Create an inference engine for TensorRT by predicting over a single image."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "B9PCp7JRxx6o",
        "outputId": "c142db41-9932-44cb-ba2d-2021b9057923"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "210 ms ± 4.24 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
          ]
        }
      ],
      "source": [
        "%%timeit\n",
        "# Inference speed with first image.\n",
        "results = list(\n",
        "    map(\n",
        "        lambda model: perform_detection(model, image1),\n",
        "        detection_fns_tensorrt\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WfO9Fp-fEGrT"
      },
      "source": [
        "## Checking speed with TensorRT model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Rx1cd0RrDmQm",
        "outputId": "bda8db46-c793-4f69-f1aa-8d5746660737"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "83.8 ms ± 985 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
          ]
        }
      ],
      "source": [
        "%%timeit\n",
        "detection_fns_tensorrt[0](image2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GB8YPex-Dno3",
        "outputId": "37509a66-0087-44b9-b9c8-e249530c9c2b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "122 ms ± 1.52 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
          ]
        }
      ],
      "source": [
        "%%timeit\n",
        "detection_fns_tensorrt[1](image2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hmiE_SMMEga4"
      },
      "source": [
        "## Conclusion"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DhcCcNOxEiNZ"
      },
      "source": [
        "Average inference speed of 1st saved model over image 2    = **169 ms**\\\n",
        "Average inference speed of 1st TensorRT model over image 2 = **83.8 ms**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ShAgOkmhEwKy"
      },
      "source": [
        "Average inference speed of 2nd saved model over image 2    = **210 ms**\\\n",
        "Average inference speed of 2nd TensorRT model over image 2 = **122 ms**"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "machine_shape": "hm",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
